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PREFACE 


Research-practice partnerships can address persistent challenges by 
producing new knowledge, building capacity, and informing action. But until 
now, we have not had a unified framework for understanding what marks 
certain partnerships as effective, let alone assessing this effectiveness. We 
hope that the dimensions and indicators provided here, drawn from the 
experiences of existing partnerships, can illuminate the common goals that 
cut across diverse types of RPPs and provide guidance on where to look for 


evidence that these goals are being met. 


Vivian Tseng, 2017 
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Introduction 


Educators in schools and districts in the United 
States are working under a great deal of pressure 
to provide high quality, equitable educational 
opportunities that support students’ development 
into responsible, successful citizens prepared for 
college and careers. The U.S. school system faces 
complex challenges that impact the attainment of 
this goal, including poverty and racial inequality, 
inequitable learning opportunities, and highly varied 
teaching quality. Many stakeholders and entities, 
including families, community members, research 
organizations, foundations, and government 
organizations are attempting to address the 
challenging issues inherent in improving the quality 


of students’ education. 


However, there are concerns about whether educational 
research can make significant contributions to these 
improvement efforts. Educational practitioners report 
that the problems addressed by research studies are 
often far removed from the world of practice and 

that the findings are frequently difficult to interpret 

and challenging to apply to specific district contexts 
(Coburn, Honig, & Stein, 2009). Even when the 
questions addressed are of import to policymakers, the 
window of decision-making has already passed by the 
time that researchers are ready to present their findings 
and recommendations (Corcoran, Fuhrman, & Belcher, 


2001). 
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Research-Practice Partnerships (RPPs) have emerged 
as a promising strategy for bridging the current gulf 
between research and practice by bringing together 
experts from both fields to address problems facing K-12 
U.S. education. RPPs have been defined as long-term 
collaborations between researchers and practitioners 
that leverage research to address persistent problems 
of practice (Coburn, Penuel, & Geil, 2013). Education 
RPPs provide the organizational structure to facilitate 
sustained collaboration between researchers and 
practitioners to improve learning opportunities for 


students. 


“RPPs have been defined as long-term 
collaborations between researchers and 
practitioners that leverage research to 
address persistent problems of practice.” 


RPPs are a relatively recent development, and there 

is currently limited literature about how to assess the 
effectiveness of these organizations. This is in part 
due to ongoing discussions about what it means for 
an RPP to be effective. Funders and RPP members 
agree that traditional ways of assessing the quality of 
a research study—such as the number of publications 
in peer reviewed research journals—do not adequately 
address critical aspects of RPP work, such as the 
development of a genuine partnership between 
researchers and practitioners or the impact of the RPP 


on the participating practice and research organizations. 
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As a consequence, RPPs have difficulty not only in 
gauging whether or not they are effective but also in 
demonstrating their effectiveness to stakeholders and 


funders. 


In this white paper, we describe a framework that 
comprises five dimensions for assessing education RPPs 
(see Table 1). This framework includes a set of indicators 
for each dimension that describes where to look for 
evidence that an RPP is making progress on a particular 
dimension of effectiveness. The indicators are intended 
to guide the development of more specific protocols and 
measures that could contribute to a body of evidence 
related to partnership effectiveness. While we intend 
this framework to be useful for funders evaluating RPP 
proposals and assessing the extent to which a funded 
RPP is accomplishing its goals, we also believe it may 

be useful to current RPP teams and to researchers 

and practitioners planning an RPP. Specifically, this 
framework can helo RPP teams develop assessment 
plans that will enable them to monitor and improve their 


work on an ongoing basis. 


We begin by describing how we developed the 
dimensions and in doing so distinguish between three 
types of RPPs. We then describe the framework, which 
comprises five broad dimensions of effectiveness that 
represent objectives shared across the three types of 
RPPs. For each dimension, we articulate /ndicators of 
progress that partnerships might use to assess their 
progress in accomplishing each objective. We conclude 
by discussing how the framework might be used and 


then consider next steps for work in this area. 


1See: https://consortium.uchicago.edu. 


HENRICK, COBB, PENUEL, JACKSON, CLARK 


Development of Dimensions 


Our first step in identifying dimensions of RPP 
effectiveness was to review the existing literature on 
education RPPs, RPPs in other fields, evaluation and 
assessment of cross-sector partnerships, and districts’ 
use of research. Reviews by Coburn and colleagues 
proved to be especially helpful in organizing RRP work 
in education in a coherent manner (Coburn, Penuel, & 
Geil, 2013; Coburn & Penuel, 2016). Coburn et al. (2013) 
defined RPPs as long-term collaborations between 
researchers and practitioners that leverage research to 
address persistent problems of practice. They distinguish 
between three types of RPPs: research alliances, design 
research partnerships, and networked improvement 


communities (NICs). 


Research alliances typically focus on a specific school 
district or a single region. For example, the Consortium 
on Chicago School Research is a longstanding alliance 
between the University of Chicago, Chicago Public 
Schools, and other local community organizations.' As 

a second example, Regional Educational Laboratories 
funded by the Institute of Education Sciences are 
typically composed of multiple research alliances that 
aim to support states and districts in their geographic 
regions in using data and research to improve academic 


outcomes for students. 


In design research partnerships, researchers and 
practitioners typically collaborate to design, study, 
improve, and scale innovations in teaching and learning. 
Most design research partnerships aim to support 
teachers’ development of specific instructional practices 
that have been linked empirically to student learning in 
a particular content area. For example, the MIST project 
housed at Vanderbilt University, included a research 


team led by researchers from several universities that 
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partnered with four large urban districts that sought 
to improve the quality of middle-grades mathematics 


instruction and students’ learning. 


Networked Improvement Communities (NICs) 

are structured collaborations between education 
professionals, researchers, and designers that aim to 
support the develooment of networks that are organized 
around a shared problem of practice (Bryk et. al, 2015). 
Educational organizations participating in a NIC commit 
to use the tools and methods of improvement science 

to gather evidence, test new ideas, and share what is 
learned across the network. The goal in doing so is to 
systematically investigate how promising innovations 
can be adapted and made to work in a range of different 


contexts. 


Hybrid RPPs that have characteristics of more than 

one of these three types have become increasingly 
common since Coburn et al. (2013) paper was released, 
particularly hybrids of design research partnerships and 
NICs. 


Our second step in identifying dimensions of RPP 
effectiveness was to develop a protocol for a first round 
of semi-structured one-hour interviews conducted with 
two to three researchers and practitioners from each 

of the three types of RPPs. We selected interviewees 
from nationally recognized RPPs who are generally 
regarded as leaders in the field. The primary intent of the 
interviews was to test and revise our initial conjectures 
about possible goals for each type of RPP, capturing 
not only the RPP leaders’ current goals but also what 
they would ideally want to accomplish. We also asked 


interviewees about indicators of progress toward each 


2See: http://vanderbi.lt/mist. 
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of their goals by asking how they would gauge whether 
they were achieving their goals. In addition, we collected 
relevant documents such as surveys, evaluation tools, 
and metrics that interviewees reported using to assess 


whether their education RPP was making progress. 


“The resulting framework is normative from 
the perspective of insiders—it reflects the 
desired goals of those currently engaged in 
RPP work, rather than descriptions of what 
RPPs have accomplished to this point.” 


We analyzed the artifacts as well as audio recordings of 
interviews to develop an initial version of the framework 
for assessing RPP effectiveness and included indicators 
of progress for each dimension. We had anticipated 

that we would need to develop separate frameworks for 
each of the three types of RPPs but identified five broad 
dimensions that appeared to capture the actual and 
aspirational goals of participants in all three types. We 
shared the draft framework with and received feedback 
from our first round of interviewees. We then sought to 
address weaknesses and ambiguities suggested by the 
feedback by conducting a second round of interviews 
with eight additional members of the RPP community, 
including three members of NICs, two funders, and three 
leaders of national, state, and local educational agencies. 
As a final step, we solicited additional feedback on 

the revised framework when we shared it at a Design 
Based Implementation Research (DBIR) workshop and 
at the National Network of Education Research-Practice 
Partnerships (NNERPP) annual meeting during the 
summer of 2016. This feedback and revision process 
helped us refine the framework to be more useful for 
funders in their assessment of RPPs and also for team 


members engaged in an RPP. 
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The resulting framework is normative from the 
perspective of insiders—it reflects the desired goals 

of those currently engaged in RPP work, rather than 
descriptions of what RPPs have accomplished to this 
point. A recent study (Farrell et al., 2017) provides 
confirming evidence that goals from all five dimensions 
are pursued by partnerships of all tyoes. The dimensions 
are, moreover, goals to which participants believe they 
should be held to account by stakeholders, including 


funding agencies. 


In what follows, we describe each dimension and the 
related indicators of progress. The first two dimensions 
pertain to the quality of the partnership. The first 
dimension focuses on the RPP goal of building trust 
and cultivating partnership relationships. The second 
dimension relates to the RPP goal of conducting 
rigorous research to inform action. Significant progress 
on these two dimensions appears to be critical if 

an RPP is to make progress on the remaining three 
dimensions, which pertain to the results achieved by 
the partnership. The third dimension concerns the RPP 
objective of supporting the partner practice organization 
in achieving its goals. The fourth dimension relates to 
the RPP goal of producing knowledge that can inform 
educational improvement efforts more broadly. Finally, 
the fifth dimension focuses on the RPP goal of bu//ding 
the capacity of participating researchers, practitioners, 
practice organizations, and research organizations to 


engage in partnership work. 


It is important to note that some of these objectives 
are more important to some types of partnerships 
than others. For example, some place-based alliances 
do not prioritize producing knowledge that can 
inform educational improvement efforts outside of 


the partnership. Although each of the five dimensions 
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relates to a specific objective of RPPs, the key objective 
of improving the use of research in practice spans 

the second, third, and fifth dimensions. The second 
dimension describes conducting research that is 

both rigorous and relevant to practitioners, and thus 
has greater potential to be used; the third dimension 
concerns the extent to which the research is actually 
used to address specific problems of practice; and the 
fifth dimension focuses on building the capacity to use 


research to address problems of practice more generally. 
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DIMENSION 1 


Building trust and 
cultivating partnership 


relationships 


All of our interviewees indicated that strong 
interpersonal relationships grounded in trust are an 
essential aspect of effective education RPPs. They 
emphasized that such relationships are an essential 
foundation for productive collaborations that address 
challenging problems in education. Our interviewees also 
indicated that developing and maintaining relationships 
of trust between members of different organizations 
(e.g. school districts, universities, state education 
departments, and research firms) is challenging and 
requires significant time and commitment. Challenges 
that can negatively impact relationship building include 
a history of limited interaction between researchers and 
practitioners, a power imbalance between researchers 
and practitioners, differences in priorities and values, 
time constraints, and conflicting views of what counts as 


evidence (Coburn, Penuel, & Geil, 2013). 


All the RPP leaders we interviewed reported that in the 
absence of strong relationships and trust, partnerships 
usually fail. For this reason, it is important to assess 
the extent to which RPPs are cultivating partnership 
relationships and fostering productive collaborations 


during all phases of their work. 
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What are indicators of progress on this 
dimension? 


INDICATOR 1 

One indicator that an RPP is making progress in building 
trust and cultivating productive relationships is that 
researchers and practitioners routinely work together. 
Investing time for joint work signifies commitment to 
the partnership and provides opportunities for team 
members to build trust, develop shared goals and work 
plans, and learn together. Our interviewees described 
meetings in which researchers and practitioners 
developed research agendas, assessed partnership 
progress, revised work plans, and adjusted partnership 
goals. Interviewees reported that the amount of time 
team members spend together can vary for a number 
of reasons, such as the physical distance between the 
research and educational organizations, the type of 
RPP, and the current focus of the work. However, as one 
interviewee aptly observed, signs that a partnership 
might be struggling include the repeated cancellation 
or rescheduling of meetings and the failure of a team to 


prioritize scheduling time to work together. 
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INDICATOR 2 

A second indicator of partnership development is that 
the RPP establishes routines that promote collaborative 
decision making and guard against power imbalances. 
The standard characterizations of researchers as 
producers of knowledge and practitioners as consumers 
and implementers involves a deeply ingrained power 
imbalance that could well be reproduced when an RPP 
brings together groups of researchers and practitioners 
who do not have a history of working with each other. 
Our interviewees emphasized the importance of 
establishing organizational structures, routines, and 
norms of interaction that support the development 

of high-functioning collaborative teams. For example, 
several interviewees described structuring partnership 
meetings so that all team members can contribute 
substantially, thereby facilitating communication, 
guarding against possible imbalances of power, and 
creating conditions for the development of trust. Our 
interviewees viewed these intentional efforts to cultivate 
settings in which it is safe to share differing perspectives 
as essential. Several interviewees also reported that 
established partnership routines helped their RPP 
weather challenges, such as research findings that raised 
unanticipated and/or politically charged issues and 
turnover in leadership positions within the educational 


organization (e.g., school district personnel). 


INDICATOR 3 

A third, closely related indicator is that RPP members 
establish norms of interaction that support collaborative 
decision making and equitable participation in all 
phases of the work. While routines reflect how meetings 
are structured, norms of interaction reflect how that 
structure is enacted. Thus, routines create conditions in 
which trust can develop, and norms indicate whether 


trust is actually being established. For example, 
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although partnership meetings might be structured so 
that all team members can contribute, in the absence 
of productive norms, this routine might become a 
gratuitous compliance ritual. Evidence of productive 
partnership norms indicates that trust emerges as team 


members fulfill their obligations to one another. 


“strong interpersonal relationships 
grounded in trust are an essential aspect of 
effective education RPPs.” 


INDICATOR 4 

A fourth indicator that an RPP is cultivating productive 
partnership relationships is that RPP members recognize 
and respect one another’s perspectives and diverse 
forms of expertise. When team members in partnerships 
believe that the work they can do collectively is higher 
quality than the work they would do separately, and 

that relations of power are fluid and follow the contours 
of expertise (Elmore, 2006), there is a much greater 
likelihood that the partnership will sustain over the long 
term. One interviewee emphasized the importance of an 
RPP team taking the time to understand the strengths 
and types of expertise that each team member can 
contribute in order to capitalize on the knowledge 
available to the group. This understanding, in turn, 
allowed the group to address a given problem by 
drawing on team members’ relevant expertise rather 
than relying on positional authority to drive the team’s 


work. 


INDICATOR 5 
A fifth indicator for this dimension is that partnership 
goals take into account team members’ work demands 


and roles in their respective organizations. Interviewees 
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reported that establishing both achievable goals 

and feasible partnership commitments supports the 
development of productive collaborative relationships 
and trust. One interviewee described this indicator in 
terms of “making sure the footprint of the partnership 
matched the partnership resources and time 
commitments.” Another interviewee described a routine 
in which participants shared their work challenges and 
successes at the beginning of team meetings, which 
helped team members better understand differences in 
their organizational contexts and consider competing 


work demands when setting partnership goals. 
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DIMENSION 2 


Conducting rigorous 
research to inform action 


Interviewees across the three types of partnerships 
agreed that most education research has little impact 
on practice for the straightforward reason that it is 

not relevant to practitioners’ concerns. They also 
characterized the development and work of RPPs 

as strategies to mobilize research to inform action 
(Weiss, 1979). Our interviewees also indicated that 
relevant criteria for high-quality research in the context 
of partnership work include both that the research 

is methodologically rigorous and that it is re/evant, 
timely, and useful for practitioners and policymakers. 
Therefore, the second dimension we identified for 
assessing RPPs deals explicitly with the quality of the 
research conducted within an RPP. This dimension 
concerns what research an RPP conducts, why the RPP 
conducts this research, and how the RPP conducts the 
research. The indicators for this dimension therefore 
span the entire research process, from the formulation of 
research questions to the collection and analysis of data 
to the sharing of findings and the clarification of their 
implications for action. Given this broad perspective, 
we should acknowledge that some of the indicators 
would not be relevant for investigations that address 

a significant problem of practice by analyzing existing 
administrative datasets. This is also the case for reviews 


and syntheses of current research literature relevant to 
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a particular problem of practice, a significant aspect of 


most if not all RPPs’ research activity. 


What are indicators of progress on this 
dimension? 


INDICATOR 1 

One indicator of progress on this dimension is that the 
RPP conducts research that addresses problems of 
practice facing the practice organization. Interviewees 
across RPP types stressed the importance of researchers 
and practitioners determining the focus of the research 
collaboratively. Alliance members described bringing 
together multiple stakeholders, including researchers, 
educators, and community members, to jointly identify 
and prioritize the problems that the RPP would address. 
Design research team members described negotiating 
with education leaders and teachers to focus the work 
around problems of teaching and learning that were 
relevant to the practice organizations and that the 
researchers had the expertise to help address. Members 
of NICs described expert convenings, in which educators, 
researchers, and designers both negotiated how focal 
problems should be framed and specified the systems 


that produced the current situation. 
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INDICATOR 2 

A second indicator of progress in conducting high- 
quality research is that the RPP has developed 
systematic processes for collecting, organizing, 
analyzing, and synthesizing data. Interviewees stressed 
that developing an efficient infrastructure for collecting 
and analyzing data was essential given that research 
findings are only useful if they are available when 


practitioners have to make critical decisions. 


“..most education research has little impact 
on practice for the straightforward reason 
that it is not relevant to practitioners’ 
concerns.” 


INDICATOR 3 

A third indicator of progress on this dimension is that 
decisions about research methods and designs balance 
rigor and feasibility. Our interviewees agreed that RPP 
research must be methodologically rigorous and that 
findings must be valid and trustworthy, as they can 
impact the education of large numbers of students. In 
this respect, RPP research is like other approaches to 
education research that aspire to rigor by employing 
systematic methods of inquiry. However, our interviewees 
also emphasized the importance of feasibility in the 
context of RPP work and described selecting the most 
rigorous yet feasible research methodology that would 
enable them to address problems of practice within a 
reasonable timeframe. The methods that interviewees 
described using varied somewhat by partnership type 
and included standard qualitative and quantitative 
techniques, design research, and improvement science 
methods. Interviewees from all three types of RPPs also 
indicated the importance of using multiple sources of 


evidence, including the perspectives of key stakeholders 
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in the partnership, to support research claims. In 
addition, they made it clear that taking the systems, 
policies, and routines of the educational organization 
into account when interpreting findings is an aspect of 


high-quality research that can inform action. 


INDICATOR 4 

A fourth indicator of progress is that the RPP conducts 
research to clarify and further specify problems of 
practice prior to identifying and assessing strategies 

for addressing those problems. Interviewees described 
taking time at the beginning of each study to better 
understand the problems of practice that the partnership 
hoped to address. These initial investigations into the 
problems were often described as a critical first step 
that led to significant learning for both the researchers 
and practitioners involved in the partnership. When 
existing research on a problem of practice was already 
available, researchers reported synthesizing and sharing 
the findings with practitioners. For example, alliance 
researchers described conducting descriptive analyses 
to provide evidence that allowed them to better 
understand the depth and complexity of an issue. Design 
researchers described conducting initial, empirically- 
grounded assessments of the current practices of 
particular role groups (e.g., teachers, coaches, principals) 
and of relevant aspects of the contexts in which they had 
developed those practices. NIC team members described 
mapping existing organizational systems to determine 
why they produced current outcomes. Across the three 
tyoes of RPPs, interviewees confirmed that reviewing 

the existent literature and conducting initial exploratory 
research allowed the team to develop a deeper, shared 
understanding of focal problems before beginning to 


investigate strategies to address them. 
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INDICATOR 5 

A fifth indicator of progress is that findings are shared 
in ways that take account of the needs of the practice 
organization. Interviewees described the importance 
of maintaining transparency when sharing findings with 
a partner educational organization, especially if the 
findings have negative implications and will be released 
to the broader community. Interviewees across all 

RPP types also reported jointly interpreting findings in 
partnership meetings to build a shared understanding 
of the research results and their implications for 

action. Additionally, many interviewees discussed the 
importance of sharing findings in a variety of ways that 
might be appropriate for different audiences, including 


research briefs, infographics, and webinars. 
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DIMENSION 3 


Supporting the partner 
practice organizationin 
achieving its goals 


Interviewees across partnership types reported that a 
primary goal of RPPs is to support the improvement 
efforts of the partner practice organization. Hence, 

the third dimension concerns the use of RPP research 
to address focal problems of practice. This stands 

in contrast to more traditional research studies that 
rarely have the explicit aim of supporting a practice 
organization’s achievement of its goals. Usually, any 
benefits of a practice organization’s participation in 

a traditional research project are viewed as a bonus 
rather than as a central goal of the project. While all our 
interviewees endorsed this goal strongly, their accounts 
of ways in which RPPs might provide support varied 
widely. We found that the type of RPP, the specific goals 
of the partnership, the expertise of team members, and 


the sources of funding influenced the types of support. 


Looking across the interviews, alliance researchers 
usually described providing additional research capacity, 
including evaluating local policies and programs and 
developing indicators to predict desired outcomes. 
Design researchers recounted co-designing and 


co-implementing innovations such as professional 
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development activities and instructional materials. NIC 
researchers described providing improvement science, 
content, and analytical expertise relevant to the problem 
being addressed; compiling and sharing findings and 
innovations across the network; and fostering the 
development of the network by organizing meetings 
and facilitating communication. Interviewees from all 
three types of RPPs also described interpreting research 
findings together (e.g., through data sharing and 
planning meetings) to support the improvement efforts 


of the partner practice organization. 


“a primary goal of RPPs is to support the 
improvement efforts of the partner practice 
organization.” 


Many reported that evidence of impact on the practice 
organization’s attainment of its improvement goals is 
especially consequential because it is frequently the 
sole dimension of effectiveness on which their RPPs 
are assessed. However, a significant majority indicated 


that their goals are not exclusively local and that they 


11 
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consider it equally important to influence practice and 
research more widely by contributing knowledge and 


insights to their respective fields. 


Interviewees consistently indicated that assessing 
impact on practice organizations’ improvement efforts 
can be challenging. The most commonly reported 
challenge concerns the influence of the local context 
on partnership work. School and district contexts 
include elements that are targets of improvement, as 
well as elements that are not likely to change. Often, 
aspects of the local context that are beyond the scope 
of partnership work strongly influence the impact of the 
partner practice organization’s improvement efforts. 
For example, high turnover in practice organizations 
can influence an RPP’s impact. Despite this and other 
challenges, it is essential for RPPs to determine feasible 
ways to assess the impact of their work on local 


improvement efforts. 


What are indicators of progress on this 
dimension? 


INDICATOR 1 

A first indicator of progress on this dimension is 

that the RPP provides research and evidence to 

support improvements in the partner organization. 
Interviewees reported an urgency to show partnership 
impact on student outcomes, even when the focus 

of the partnership work was several steps removed 

from student outcomes (e.g. principal instructional 
leadership). One interviewee described addressing this 
issue by documenting evidence of how the partnership 
activities supported key aspects of the district’s strategic 
plan. Other interviewees indicated the value of collecting 
data on relevant proximal and distal outcomes (e.g., 


the quality of classroom instruction and the quality of 
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the feedback that principals provide on instruction). 
This approach suggests that in cases where student 
outcomes are the ultimate evidence of partnership 
success, it is also important for RPPs and funders to 
assess incremental impacts of partnership work. For 
example, a partnership might focus on improving school 
leaders’ practices as instructional leaders. In this case, 
the claim that any gains in student achievement are a 
consequence of partnership work is almost certainly 
flawed in light of other initiatives that the district 

might be implementing (e.g., teacher professional 
development, teacher learning communities, coaching, 
revisions to district curriculum frameworks, adoption of 
new curriculum materials, etc.). The RPP might therefore 
document evidence of improvements in the quality 

of feedback that principals give to teachers and also 
investigate whether, jn the context of the other local 
initiatives, higher quality principal feedback supports 


improvements in the quality of teaching. 


INDICATOR 2 

A second indicator of progress on this dimension is 
that the RPP helps the practice organization identify 
productive strategies for addressing problems of 
practice. Our interviewees consistently indicated that 
RPP research can inform the practice organization’s 
identification of productive organizational policies, 
processes, and structures, as well as productive 
strategies for improving leadership and teaching 
practices. The approaches they described included 
conducting research studies to assess the effectiveness 
of current strategies and reviewing relevant research 
literature to identify potential evidence-based 


improvement strategies to implement. 
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INDICATOR 3 

A third indicator of progress on this dimension is that the 
RPP informs the practice organization’s implementation 
and ongoing adjustments of improvement strategies. 
Interviewees, particularly from NICs and design research 
partnerships, reported using research evidence to 
understand how current improvement strategies were 
being implemented and how to adjust the strategies to 


make them more effective. 
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DIMENSION 4 


Producing knowledge 
that can inform 
educational improvement 
efforts more broadly 


It is clear from our interviews that most RPPs 

are working not only to support partner practice 
organizations in achieving improvement goals, but 
also to influence practice and research more broadly. 
The minority of RPPs for which influencing practice 
and research more broadly was not a goal tended to 
be place-based alliances supported by local funders 
who had fewer expectations for publishing in peer- 
reviewed academic journals or presenting findings at 
research conferences. When assessing most RPPs, it is 
therefore important to attend to the fourth dimension of 
effectiveness, which concerns impact on research and 


practice beyond the partnership. 


Our interviews indicate that the ways in which an RPP 
attempts to make broader contributions vary depending 
on the type of RPP. For example, alliance interviewees 
described sharing indicator systems for important 


outcomes, such as the on-track indicator 


3See http://learndbir.org/. 
4See http://carnegiefoundation.org/. 
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systems for high-school graduation developed by the 
University of Chicago Consortium on School Research 
and the Research Alliance for New York City Schools 
(Kemple, Segeritz, & Stephenson, 2013). Design 
research interviewees discussed designs for supporting 
professional learning in specific content areas (e.g., 
designs for supporting the development of science 
coaches), as well as content- and practice-specific 
principles for designing supports for professional 
learning that can be used by other partnerships.* NIC 
interviewees reported sharing improvement science tools 
and practices across networks, as well as developing 
“change packages” that offer productive strategies for 
addressing problems of practice and are adaptable for a 


range of different contexts.4 
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What are indicators of progress on this 
dimension? 


INDICATOR 1 

A first indicator of progress toward impacting research 
and practice is that the RPP develops and shares 
knowledge and theory that contributes to the research 
base. A common indicator that an RPP is informing the 
work of others is the extent to which the partnership 
develops and shares findings and theory that furthers 
the field’s understanding of what it takes to support 
educational improvement. Common vehicles for making 
research contributions that can inform educational 
improvement efforts include articles in peer-reviewed 
research journals, books, conference presentations, 
workshops, websites, and reports. Interviewees 
reported tracking a range of activities to document 

the contributions of their RPP to research outside of 
the partnership, including tracking typical research 
dissemination activities, such as research presentations 


and publications. 


“the ways in which an RPP attempts to 
make broader contributions vary depending 
on the type of RPP.” 


INDICATOR 2 

A second indicator of progress on this dimension is that 
the RPP develops and shares new tools and/or routines 
that can be adapted to support improvement work in 
other settings. The range of indicators and routines 
produced by RPPs is broad and includes indicators that 
students’ learning is on track, that students feel they 
belong in a particular class, and that teachers have 
reason and motivation to improve their instructional 


practices. Examples of routines include those for 
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productive collaboration between researchers and 
practitioners, for conducting formative assessments, and 


for testing adjustments to a practice, process, or tool. 


INDICATOR 3 

A third indicator of progress in this dimension is that 
the RPP develops two dissemination plans, one that 
supports partnership goals (e.g., sharing findings 

with community members) and a second for broader 
dissemination. For example, RPP researchers might 
produce memos and reports to share findings with their 
practice partners, journal articles to share findings with 
a broader research audience, and blog posts to share 


findings with a broader practice audience. 
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DIMENSION 5 


Building the capacity 

of participating 
researchers, practitioners, 
practice organizations, an 
research organizations 


to engage in 


partnership work 


The viability and sustainability of RPPs requires the 
building of human and organizational capacity to 
conduct partnership work. Interviewees reported that 
engaging in partnership work requires a fundamental 
shift in professional identity and in research practices. 
At present, a relatively small number of education 
researchers participate in RPPs, and the number 
engaged in RPP work that focuses on the quality of 
classroom instruction and on equity in students’ learning 
opportunities is even smaller. As a consequence, 


opportunities for doctoral students and junior scholars 
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to learn how to initiate and participate productively in 

an RPP are limited, even though there is an increasing 
demand from junior scholars for such opportunities. At 
the same time, most practitioners have few opportunities 
to engage in collaborative work with others professionals 
outside their organizations or even to develop a sense 

of what a genuine partnership with researchers might 
look like. Furthermore, few formal teacher education and 
administration programs prepare practitioners for this 


potential aspect of their work. 


16 


ASSESSING RESEARCH-PRACTICE PARTNERSHIPS: FIVE DIMENSIONS OF EFFECTIVENESS 


Traditionally, researchers and practitioners have 
engaged in largely separate professional activities. 

As a consequence, collaborating with each other 

on common problems requires significant changes 

in their respective practices, as well as in how each 
group views the other. For example, it is essential that 
RPP researchers move beyond viewing educational 
organizations as sites to be used for investigating 
problems that research communities consider important, 
and instead come to view these organizations as sites 
of problems of practice that need to be investigated 
with practitioners. This requires that researchers take 
the time to understand the challenges that practitioners 
face and that, in clarifying the nature of the challenges, 
they come to view the resulting problems as their 
problems. Our interviewees indicated that collaborating 
to negotiate the focus points of joint work and to 
develop and implement improvement initiatives could 
be opportunities for both researchers’ and practitioners’ 
learning. For example, graduate students’ involvement 
in this work can enable them to become conversant in 
multiple methodologies, whereas conventional graduate 
education often emphasizes proficiency in a single 
methodology. For their part, participating in an RPP 
can support practitioners’ development in designing 
and implementing improvement initiatives and in using 
research and data to inform all phases of improvement 


work. 


In addition, several interviewees reported that their 
RPPs aim to support the development of the norms 
and culture of their partner educational organizations 
around the use of research and other evidence. Other 
interviewees indicated that their RPPs aim to build the 
organizational capacity of partner organizations by 
supporting the establishment of organizational routines 


for monitoring and addressing the problems that almost 
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inevitably arise during implementation of potential 


solutions in a systematic, data-driven manner. 


“it is essential that RPP researchers move 
beyond viewing educational organizations 
as sites to be used for investigating problems 
that research communities consider 
important, and instead come to view 

these organizations as sites of problems of 
practice that need to be investigated with 
practitioners.” 


The fifth dimension for assessing RPPs is therefore the 
extent to which an RPP supports the development of 
team members’ capacities to productively engage in 
partnership work. By attending to this issue explicitly, 
RPPs can support both researchers and practitioners in 
developing new capacities and ways of working, while 
also helping partner educational organizations establish 
routines around the use of evidence in order to inform 


improvement work. 


What are indicators of progress on this 
dimension? 


INDICATOR 1 

A first indicator of progress toward increasing team 
members’ capacity to conduct partnership work is that 
team members develop professional identities that 
value engaging in sustained collaborative inquiry with 
one another to address persistent problems of practice. 
Interviewees described working to develop a culture 
that reflects the belief that the work researchers and 
practitioners can do together is better than the work 
they would be able to do separately because their 


perspectives and areas of expertise are complementary 
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and produce novel solutions to challenging problems 


when brought together. 


INDICATOR 2 

A second indicator of progress is that team members 
assume new roles and develop the capacity to conduct 
partnership activities. This often necessitates that 
researchers and practitioners develop new professional 
identities, practices, and ways of working. Interviewees 
described the importance of supporting researchers’ 
development of the interpersonal, organizational, and 
communication skills needed to engage productively in 
typical partnership activities and to engage effectively 
with partnership stakeholders. Interviewees indicated 
that these capabilities are crucial when negotiating 

a shared focus with practitioners, coordinating the 
work within and across organizations, and sharing 
findings with different audiences. The interviews also 
suggest that there might be differences in some of 

the capabilities required for the three types of RPPs. 
Alliance researchers described the skills needed to 
effectively respond to the political nature of findings and 
negotiate data sharing agreements. Design researchers 
described interpersonal and design skills that are integral 
to collaborative design in an RPP. NIC researchers 
described facilitation and collaboration skills necessary 
for developing professional norms of trust to share 
innovations, data, and findings across a network and 
for supporting problem-solving conversations that are 
grounded in data and involve the use of improvement 


science methods and tools. 


INDICATOR 3 

A third indicator of progress on this dimension of RPP 
effectiveness is that the participating research and 
educational organizations provide capacity-building 


opportunities to team members. The researchers 
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we interviewed described providing opportunities 

for graduate students and post-doctoral fellows to 
participate in and eventually play a leadership role in 
various aspects of the partnership. For their part, the 
practitioners described inviting colleagues to participate 
in partnership activities that were relevant to their job 


responsibilities. 


INDICATOR 4 

A fourth indicator of progress is that the work of 

the RPP contributes to a change in the practice 
organization’s norms, culture, and routines around the 
use of research and other evidence. Several interviewees 
indicated that the goals of their RPPs were not limited 
to addressing particular problems of practice but also 
included supporting members of their partner practice 
organizations to interpret and use research when making 
policy decisions and when designing and implementing 
improvement initiatives more generally. Interviewees 
described aiming to support the development of a 
culture of research use within educational organizations, 
something that past studies have found is associated 
with a higher level of research use (Penuel, Briggs, et al., 
2017): 


INDICATOR 5 

A fifth indicator of progress on this dimension is 

that there are shifts in professional expectations 

for education researchers and for practitioners that 
reward members from each organization for sustained 
participation in significant partnership work. For 
example, the participation of practice organizations’ 
leaders in partnership activities serves to indicate the 
importance of the work. Similarly, adjustments in the 
criteria for tenure and promotion of educational research 
faculty to include partnership activities signals that RPP 


work is valued. 
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INDICATOR 6 

A sixth indicator of progress on this dimension of 
effectiveness is that the RPP establishes conditions in 
the practice organization that lead to sustained impact 
beyond the life of the partnership. Some interviewees 
reported documenting the spread of activities, or the 


extension of practices first initiated by partnership 


activities that move beyond the scope of the partnership. 


Other interviewees collected evidence of sustainability 
by documenting partnership routines, tools, and 
vocabulary that continued either after a partnership 


ended or after the focus of partnership work changed. 


INDICATOR 7 

A seventh indicator of progress is that research and 
educational organizations allocate resources to 
support partnership work. For example, restructuring 
job responsibilities of practitioners to provide time for 
collaboration with researchers can impact the common 
view that partnership activities are “in addition” 

to regular work responsibilities, as does research 
organizations’ funding of personnel to faciltiate the 


establishment of partnerships with practitioners. 
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Using the framework 


The five dimensions of RPP effectiveness that we have 
discussed are intended to be broadly applicable across 
education RPPs. Two issues need to be addressed 

when using the framework to guide the assessment of a 
particular RPP. The first concerns the customization of 
the framework to the goals and theory of improvement 
of the RPP, and the second concerns the development of 
measures and protocols that serve to operationalize the 


indicators that are relevant to a particular RPP. 


Customizing the Framework 


The dimensions of effectiveness that cut across the 
three types of RPPs are relatively general. When 
designing an assessment of a specific RPP, however, 
other characteristics of the partnership, which are not 
adequately captured here, must be considered. Key 
characteristics that should be taken into account include 
the goals of the RPP, as RPPs differ in terms of what 
they are attempting to improve (e.g., the coordination 
between key departments of the practice organization, 
coaches’ practices while working with teachers, student 
mathematics achievement on an assessment aligned 
with college and career readiness standards). In addition, 
it is also important to tailor the assessment to the RPP’s 
theory of improvement (Tol), which articulates the 
hypothesized mechanisms or processes by which the 


intended improvements will be produced. 
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A key step in designing a specific assessment is 

to identify the indicators for each dimension of 
effectiveness that are relevant to the RPP and customize 
those indicators to the RPP’s goals and Tol. As an 
illustration, we focus on the five proposed indicators for 
the first dimension of effectiveness, bui/ding trust and 


cultivating partnership relationships: 


e Researchers and practitioners routinely work 
together. 

¢ The RPP establishes routines that promote 
collaborative decision making and guard against 
power imbalances. 

« RPP members establish norms of interaction 
that support collaborative decision making and 
equitable participation in all phases of the work. 

¢ RPP members recognize and respect one another’s 
perspectives and diverse forms of expertise. 

¢ Partnership goals take into account team members’ 
work demands and roles in their respective 


organizations. 


These indicators describe where to look for evidence 
that an RPP is making progress on a particular dimension 
of effectiveness. However, they are generic and have to 
be fleshed out by customizing them to the Tol of the 
RPP being assessed. Consider, for example, the first of 
these indicators: researchers and practitioners routinely 
work together. Clearly, if the work is to contribute to the 


development of productive collaborative relationships, it 


20 


ASSESSING RESEARCH-PRACTICE PARTNERSHIPS: FIVE DIMENSIONS OF EFFECTIVENESS 


is important that both researchers and practitioners view 
the work of the RPP as contributing to the attainment 

of an improvement goal. However, the significant work 
that has to be accomplished fluctuates over time in most 
RPPs. The assessment of an RPP’s progress on the first 
dimension therefore needs be customized to the ebb 


and flow of the RPP’s work demands. 


“it is important to tailor the assessment 
to the RPP’s theory of improvement, which 
articulates the hypothesized mechanisms 
or processes by which the intended 
improvements will be produced.” 


To this point, we have spoken as though an RPP has a 
single, overarching Tol. However, the above illustration 
clarifies that it can be useful to view RPPs as having 

a Tol for each of the five dimension of effectiveness, 
such as, for example, building trust and collaborative 
relationships. Although it might seem burdensome to 
articulate a Tol for each dimension, this is often a useful 
exercise, as it can lead to revisions of a previously largely 
implicit Tol, revisions that have the potential to improve 


the effectiveness of the RPP on the relevant dimension. 


We should acknowledge that in explicating the Tol for 
each dimension, it might become apparent that some of 
the indicators we have proposed are less important for a 
particular RPP. In addition, other indicators that we have 
not considered might be relevant to a particular RPP. 
The identification of additional indicators can inform 

the elaboration of the framework. Such revisions are 
consistent with our view of the framework as a work in 


progress that is best improved as it is used. 
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Developing Measures and Protocols 


The indicators included here, which describe where to 
look for evidence that an RPP is making progress ona 
particular dimension of effectiveness, are intended to 
guide the development of more specific quantitative 
measures and qualitative protocols. Two tradeoffs need 
to be considered when developing such measures and 


protocols. 


The first is the tradeoff between general assessment 
tools that can be used across RPPs and boutique 

tools developed to assess a particular RPP. 

General assessment tools support comparison and 
benchmarking but might not be a sufficiently good fit 
with the corresponding indicator once they have been 
customized for a particular RPP. For example, a survey 
assessment instrument developed by Wentworth, 
Mazzeo, and Connolly (2017) to assess the impact 

of an RPP on practitioners’ use of evidence in their 
decision making has proven useful across a range of 
RPPs. Additionally, Penuel and colleagues have shared 
scales for assessing a practice organization’s culture 

of research use and its conceptual use of research 
(Penuel, Briggs et al., 2017). In contrast, boutique tools 
are designed to fit with the corresponding customized 
indicators but require resources to develop and might 
well have limited broader utility. We speculate that it will 
be easier to use common assessment tools for some of 
the five dimensions of effectiveness than for others. For 
example, the indicators for the first dimension, building 
trust and cultivating partnership relationships, might 

be applicable to most if not all RPPs with only minor 
adjustments when they are customized. In contrast, 

the indicators for the third dimension of effectiveness, 
supporting the partner practice organization in achieving 


its goals, will need to be adjusted to a partnership’s 
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specific improvement goals, and these goals vary widely 
across RPPs. Furthermore, RPPs may also need to gauge 
progress toward improvement goals that emerge while 

their work is in progress, thus requiring the appropriation 


or development of additional assessment tools. 


In our view, existing assessment tools should be 
examined for their fit with the corresponding customized 
indicators before embarking on the development of 
boutique tools. We therefore see considerable value ina 
repository of measures and protocols that can support 
the use of common assessment tools. A repository of 
measures can also make boutique tools available to the 
possibly small number of RPPs to which they might be 
useful. A repository of this tyoe might be housed by a 
national organization for RPP work, such as the National 
Network of Education Research-Practice Partnerships 
(NNERPP), or by a funder whose interests and portfolio 
include RPPs. 


The second tradeoff happens as RPPs decide whether 
to develop and use quantitative measures or qualitative 
protocols in assessing their progress on each dimension. 
Quantitative measures such as short surveys can 
usually be administered and analyzed relatively quickly 
with only minor disruptions to practitioners’ ongoing 
work. However, developing quantitative measures to 
assess the quality of some aspects of practitioners’ 

and researchers’ individual and joint work is likely to 

be extremely challenging, especially when there is no 
common language for describing particular domains 

of activity. In this latter case, the tradeoff might well 
favor using qualitative protocols even though additional 
resources are needed to collect and analyze data and 
the assessment process might be more disruptive for 
practitioners. Looking across the five dimensions of 
effectiveness, we speculate that it might be relatively 
straightforward to develop short surveys for most if 


not all of the indicators for the fifth dimension, bui/ding 
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the capacity of participating researchers, practitioners, 
practice organizations, and research organizations to 
engage in partnership work. In contrast, it might be more 
challenging to develop quantitative measures for most of 
the indicators for the second dimension, conducting and 


using high-quality research to inform action. 


“In addition to considering the feasibility 
of developing solid quantitative measures, 
reasonable resolutions to this tradeoff 
require that the purpose of and audience for 
the assessment be taken into account.” 


In addition to considering the feasibility of developing 
solid quantitative measures, reasonable resolutions to 
this tradeoff require that the purpose of and audience 
for the assessment be taken into account. For a funder 
that wants to assess the RPPs in its portfolio, or for an 
RPP that wants to demonstrate its accomplishments to 
local stakeholders or funders, relying on quantitative 
measures might suffice, provided the resulting picture 
of the partnership’s effectiveness, while incomplete, 

is not misleading. However, for an RPP that wants to 
use an assessment to improve its effectiveness, it is 
likely necessary to use qualitative protocols as well as 
quantitative measures. In this latter case, data collection 
and analysis can be made manageable by focusing on 
particular dimensions and indicators during various 
phases of the RPP’s lifespan. For example, it is critical 
that researchers and practitioners establish productive 
collaborative relationships during the initial phase of an 
RPP. For an RPP that wants to improve its effectiveness 
in this regard, it is probably necessary to use a mixed- 
methods approach to assess the progress it is making on 


the first dimension of effectiveness, as resources permit. 
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Conclusion 


We have proposed a framework for assessing RPPs that 
specifies five dimensions of effectiveness and includes 
associated indicators of progress on each dimension. 

We speculate that the dimensions are reasonably solid 
given the relatively large number of RPP participants and 
other stakeholders who have reviewed the framework. 

In contrast, we regard the delineation of indicators as 

a work in progress and have suggested that additional 
indicators might be identified when the framework is 
actually used to guide the assessment of RPPs. We also 
discussed two important steps in using the framework. 
First, we emphasized the importance of articulating a Tol 
for each dimension of effectiveness and of customizing 
the indicators to this Tol. Second, we took a pragmatic 
approach when discussing the develooment of measures 
and protocols that operationalize the customized 
indicators by clarifying the tradeoffs in using general or 
boutique assessment tools and in deciding to develop 
and use quantitative measures or qualitative protocols to 


assess progress on each dimension. 


We noted at the beginning of the paper that RPPs area 
promising strategy for making substantial contributions 
to the improvement of students’ education by harnessing 
researchers’ and practitioners’ complementary forms of 
expertise. However, the development of feasible, solid, 
and reliable approaches for assessing and improving the 
effectiveness of RPPs is essential if that promise is to be 


realized. 
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TABLE 1 


FRAMEWORK FOR ASSESSING RESEARCH-PRACTICE PARTNERSHIPS 


DIMENSION 


Building trust and cultivating 


partnership relationships 


DIMENSION 


Conducting rigorous research to 


inform action 


DIMENSION 


Supporting the partner practice 


organization in achieving its goals 
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INDICATORS 


Researchers and practitioners routinely work together. 

The RPP establishes routines that promote collaborative decision making 
and guard against power imbalances. 

RPP members establish norms of interaction that support collaborative 
decision making and equitable participation in all phases of the work. 
RPP members recognize and respect one another’s perspectives and 
diverse forms of expertise. 

Partnership goals take into account team members’ work demands and 


roles in their respective organizations. 


INDICATORS 


RPP conducts research that addresses problems of practice facing the 
practice organization. 

The RPP establishes systematic processes for collecting, organizing, 
analyzing, and synthesizing data. 

Decisions about research methods and designs balance rigor and 
feasibility. 

The RPP conducts research to clarify and further specify problems of 
practice prior to identifying and assessing strategies for addressing those 
problems. 

Findings are shared in ways that take account of the needs of the 


practice organization. 


INDICATORS 


The RPP provides research and evidence to support improvements in the 
partner organization. 

The RPP helps the practice organization identify productive strategies for 
addressing problems of practice. 

The RPP informs the practice organization’s implementation and ongoing 


adjustments of improvement strategies. 
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DIMENSION INDICATORS 

Producing knowledge that can e The RPP develops and shares knowledge and theory that contributes to 
inform educational improvement the research base. 

efforts more broadly * The RPP develops and shares new tools and/or routines that can be 


adapted to support improvement work in other settings. 
¢ The RPP develops two dissemination plans, one that supports partnership 


goals and a second plan for broader dissemination. 


DIMENSION INDICATORS 

Building the capacity of * Team members develop professional identities that value engaging in 
participating researchers, sustained collaborative inquiry with one another to address persistent 
practitioners, practice organizations, problems of practice. 

and research organizations to * Team members assume new roles and develop the capacity to conduct 
engage in partnership work partnership activities. 


e Participating research and educational organizations provide capacity- 
building opportunities to team members. 

¢ The work of the RPP contributes to a change in the practice 
organization’s norms, culture, and routines around the use of research 
and evidence. Research and practice organizations allocate resources to 
support partnership work. 

¢ There are shifts in professional expectations for education researchers 
and for practitioners that reward members from each organization for 
sustained participation in significant partnership work. 

¢ The RPP establishes conditions in the practice organization that lead to 
sustained impact beyond the life of the partnership. 

e Research and educational organizations allocate resources to support 


partnership work. 
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